NI Computational Intelligence in Games 


Daniele Loiacono and Mike Preuss 
GECCO, July 6-10, 2013, Amsterdam, The Netherlands 


Copy ht is held by the author/owner(s). 
GECCO"13 Companion, July 6—10, 2013, Amsterdam, The Netherlands. 
ACM 978-1-4503-1964-5/13/07. 


Introduction 


Part I 


Q What is Computational Intelligence and Games about? 


Q What are the opportunities for Evolutionary Computation methods? 


Q The industry connection 


Part II 


Q Games as testbed 
Q Developing better games 
ū Developing innovative games 


Part III 


Q Competitions and available software 
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Beginnings I: gaming Beginnings II: computer gaming 


"3000 BC Dice, Senet "1961 Spacewar! - first computer video game 
2300 BC Go 1971 Galaxy Game - first arcade video game 
500 AD Chess 1972 Magnavox Odyssey console 
ca. 1600 Modern sports games 1973 Game theory: Evolutionary stable strategies 
ca. 1800 Poker, Bridge 1978-81 Space Invaders, PacMan, Donkey Kong 
1871 Pinball 1983 I, Robot - first commercial 3D video game 
1992 Wolfenstein 3D - popularization of FPS (first 


ca. 1935 Monopoly, Scrabble 


person shooters) 
1943 Game theory beginnings 


1997 Ultima Online - first massive multiplayer 
1959 Diplomacy online (MMO) game 
1997 Deep Blue beats Garry Kasparov 
1999 Blondie24: Playing Checkers by means of CI 
2006 Wii 
2008 Checkers solved 
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Beginnings III: A field forming Computer Games: trends and problems 


Q 1999: Blondie24, Learning checkers with CI and 
human players 

Q GECCO before 2005: max 2 papers/year 

Q 2005 first Computational Intelligence in Games 
(CIG) conference 

ū GECCO after 2008: around 10 papers/year 

Q IEEE TCIAIG Journal (Transactions on CI and 
Artificial Intelligence in Games) since 2009 

IEEE TRANSACTIONS ON O EvoGames track in Evo* since 2009 

anp at ix cawis ^ | O 2012: first Dagstuhl seminar on AI and CI in 

: Games 

Q Many “neighbor” conferences, etc. AIIDE, FDG, 

ines DA gameai conf. (not strictly CI, but CI welcome) 

| tr n" Q) General approach is target oriented, not 

technique oriented 


Q About 40 years of development: 
» From simplest graphics to virtual reality 
» Games use the current hardware potential 
» Graphis still dominate public perception of 
games, AI unimportant 
» Game production consists of: game 
design, storyline design, game mechanics, 
level design/content creation, character 
design, physics, playtesting etc. 
» Often teams of 50+ people for several 
years 
Q Problems: 
» Complex game realities require complex 
AI behavior to achieve Believability 
» Complex game worlds need huge effort to 
create content 


IEEECIG 2012 
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Believability 


Q Board game AI already quite good 
» Deep blue (IBM) beats Kasparov 1997 
» Checkers solved in 2008 (Schaffer) 
» Monte Carlo Tree Search (MCTS) has huge 
impact on e.g. Go AI 
Q More challenges in other games: 
» Believable appearance and behavior of all 
game components 
» NPC are a major problem (therefore 
MMOG) 
» Should act intelligently (or create this 
impression) and react appropriately 
» Must not reveal their identity by means of 
stupid mistakes (e.g. behavior loops) 


Standard game AI approaches 


big | Q Game industry prefers well known 
ET it techniques 
i K 


» Scripting 
» Rule based systems 
» Finite state machines (also hierarchical) 
» New: behavior trees 
Q Industry cautious concerning dynamics 
and non-determinism 
» What will we get? 
» How can we control game flow? 
Q Current development very dynamic, e.g. 
look at: http://aigamedev.com/ 
QO However, most current CIG research 
goes unnoticed by industry 
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Authenticity 


Q Some standard game AI problems 
example: Gothic 3 


» Path finding ineffizient, unrealistic 
paths 

» Interaction of game ai and physics 
engine: mimics, gestures, movements 

» Camera movement (e.g. following 
head but not entering the same room) 

» Again: Repetitions (game AI always 
reacts in the same way) 

Q Problem is tackled by modularization: 
Middleware 

» Specialized physics engines 

» Complex character modelling e.g. with 
EkiOne (emotions) 

» Difficulty: We may only use about 10% 
CPU-time for the whole AI 


a i dei 


: =s 


E 


Research trees 


QO Research approaches games (mainly) from 3 directions 

» Specialized algorithms: Exact algorithms or heuristics, 
e.g. applied to path finding (A*) 

» The 'classic' (deterministic) AI approach: General game 
playing (game description language GDL), tree search, 
also support vector machines (SVM) and reinforcement 
learning, strong in board games 

» Computational Intelligence (CI): Evolutionary algorithms, 
fuzzy logic, artificial neural networks, swarm intelligence 
etc., often applied for complex black-box controllers, 
analysing data 


Q However, there are overlaps. . . 


[| 
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Why shall we apply CI (evolutionary) What is the use of CI (evolutionary) 


methods to games? methods in games research? 
Q Contrary to board games, Lucas/Kendall 2006 "Evolutionary Computation and 
» Game trees often not applicable Games" (IEEE Computational Intelligence Magazine) 


» Incomplete information 

» Concurrency: During planning phase, the 
game situation changes 

» Quantifying a game situation is not trivial 

1. Good testbed to apply our methods 


> Good and fast approximations are needed 2. Do things in a better way 
3. Do things we (or they) could not do before 
Q Evolutionary Optimization is 
» Versatile, flexible, still works (somehow) 
» Copes with noise and strange search spaces 
» Can be asked to deliver a result at any time 


GECCO, July 6-10, 2013, Amsterdam, NL MINNS 3 GECCO, July 6-10, 2013, Amsterdam, NL MENS 3 


Why games are good testbeds? 


Unbiased 
More complex than academic benchmarks 
Challenging requirements (e.g., real-time) 
Cheaper than real-world problems 


Human interaction 


Games as testbeds 
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Car Setup Optimization Competition 
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Car Setup Optimization: Which parameters? 


a A car presents many parameters that can be optimized: 


» Gear ratios 

» Rear/Front wing angle 
» Brakes 

» Rear differential 

» Rear/Front anti-roll bars 


» Wheels 
* Ride 
e Toe 
e Camber 
» Suspensions T 
e Spring A 
* Bell crank 
NE d 
Camber 
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Car Setup Optimization: Overview 


Q The goal is finding the best car setup on three unknown 
tracks 
Q Challenges 
» Limited amount of time for evaluations 
» Accuracy-time tradeoff in the evaluation 
» Fake parameters that increase the search space 
» No prior knowledge 
» Car can get damages 


Car Setup Optimization: Framework 


Real Parameter Vectors 


Evaluation length 
4 


r 
Problem Definition 


Fitness evaluation 


Car Setup Optimization: Results 


Q Organized at GECCO 2009 and at Evo* 2010 
» In 2009 won by Versari et al. (PSO) 
» In 2010 won by Munoz et al. (MOEA) 


http://vimeo.com/10870222 


Physical Travelling Salesman Problem 


Q Extends the well-known 
Travelling Salesman 
Problem 
Q Add a physical dynamics to 
the movements of the 
salesman 8 
Q Solution consists of a long http://youtu.be/xV4DapXNgPE 
sequence of force vectors 
Q Run for the first time at 
GECCO 2005 and now WCCI 
2012 and at CIG 2012 
» So far best entries are 
based on MCTS and A* 
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Physical Travelling Salesman Problem 


Games as testbed: a closing balance 


QO Game-based testbeds became very popular for several 
reasons: 


» challenging 
» entertaining 
» benchmarks not just game 
Q On the other hand... 
» often just a gamification of benchmarks 


» not easy to transfer obtained knowledge on a specific 
testbed 


» the need to defend games research is shrinking 
Q Trends 


» testbed more relevant for the game research (e.g., 
believability) 


» add humans in the loop 


Why EC can improve games? 


Improve the poor AI in games 


Reduce the development cost/time 


Allow knowledge-free AI development 


Developing better games 


Evolutionary Design of NPC 


Why EC can't improve games? 


Q Early works in the field focused on 
beating the game... 


Q ... now focus is more on non-player 
characters (NPC), i.e., characters not 


Imptrewve-the-coo-ALin-games controlled by the player (either 


Nowadays AI is often very good opponents or an allies) 
Q Design choices 


» How to represent the NPC? 


" d » How compute fitness? 
AI development is not the most expensive task > Which evolutionary techniques? 
Allow knowledge-free Al development QO Some examples 


» Evolving Quake III bot 


A black-box AI design means often boring games ; mr es 
» Evolving Racing Lines in Games 
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Evolving bots for Quake III 


Evolving bots: representation 


QO How to represent the game environment? 
» Collect information with raycasting 
» Discretize local area around the NPC 


Discrete 
Representation 


Game 
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Evolving bots for Quake III 


Evolution of Reactive Rules in Multi-Player Computer Games 
Based on Imitation, Priesterjahn et al., 2005. 


http://youtu.be/mKdli9BM RI 


Evolving bots: representation (2) 


QO How to represent an NPC strategy? 
» population of if-then rules 
» game environment is matched against the rules 
» rule with the closest matching is applied 


«i» e$! 


Evolving bots: approach 


Q How to find the best rules? 
» Real-players data used to build a rule-base 
» Individuals are generated by selecting a 
random set of rules from the rule-base 


» GA is applied to evolve the best set of rules 
» Recombination works on the sets of rules 

» Mutation works on the single rules 

» Fitness is computed as 


Evolving Racing Lines for Racing Games 


fitness 2 damage dealt - damage received 
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Racing Liens: the problem Racing Lines: standard approach 


Shortest path 
or > 


minimum curvature ? Éx 
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Racing Lines: evolutionary approach 


Decoding 


Selection 
Recombination 
Mutation 


9 


Evaluation Simulation 
and 
Replacement 


—" 


ee 


POLIMI 
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Evolutionary design of NPC: 
Representation 


Q Parameterized strategy 
» requires strong domain knowledge 
» prevents emergent behaviors 
» easy to optimize and reliable 
Q Rules or trees 
» requires discrete actions or well defined basic behaviors 
» allows to integrate existing knowledge 
» allows some emergent behaviors 
Q Decision function (e.g., NN) 
» very few domain knowledge required 
» difficult to integrate existing knowledge 
» definitely allows emergent behaviors 
» might lead to unreliable results 


a POLIMI 
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What do we learn from the literature about 
evolutionary design of NPC? 
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Evolutionary design of NPC: 
Fitness function and technique 


Q Fitness function 
» generally based on in-game statistics 
» Cost/significance trade-off 
» Often noisy or non-deterministic 
Q Evolutionary technique depends on the representation used 
» Parameterized strategy ES, GA, PSO, etc. 
» Rules or trees LCS, GP, EP, etc. 
» Decision function > Neuroevolution 
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Evolutionary design of NPC: 
Expect the unexpected 


Q EvoStar 2011: Mr Racer bot (Quadflieg et al.) good but suffers from 
default clutch control 


» First approach similar to winner's clutch control: speed based 
» Autopia (winner) closes clutch below 70 km/h 

» We adapt closing (logistic) function with a bit more freedom 
» Result: using the clutch until 180 km/h is profitable 

» We would be much worse with restriction to 70 km/h 


Besides NPCs... 


http://youtu.be/Kk1mC6mZjVc 
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Besides evolutionary design of NPCs 


2 Several applications of CI to games 
» believability 
» adaptivity and in-game learning 
» analysis of player behaviors 
» improving game components 
Q In particular, notable applications of EC includes 
» evolving believable NPCs with MOEA (Togelius et al.) 
» real-time neuroevolution in games (NERO, Stanley et al.) 


» neuro-evolutionary preference 
learning (Yannakakis et al.) 


» automatic camera control (Burelli et al.) 


Developing innovative games 
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Evolving Game Content 


Q Challenges: 
» How to represent the content? 


» Which is the best representation to be evolved with 
genetic algorithms? 


» How to evaluate the game content? 


a Case studies: 
» Evolving Starcraft and Maps 
» Evolving tracks for racing games 
» Evolving maps for First Person Shooters 


Multi-objective tradeoff exploration 


Q Starcraft Maps (Togelius et al., 2010) 

» 8 objectives: base location, 
ressource fairness, choke points 
etc. 

» Unclear which objectives make 
sense 

» But single objectives can be 
discussed with users 

» We enforce formalization 

» Innovation: users may be wrong 
(e.g. fair and asymmetric maps) 

» Exploration via multi-objective 
optimization: conflicts, tradeoffs 

» This example: PCG, other uses 
similar 


Multi-objective optimization in games 


Q Great potential (> detect tradeoffs), few uses: 


» Schrum/Miikkulainen 2008: constructing complex NPC 
behavior 


» Agapitos 2008: generating diverse opponents 

» Togelius et al. 2010: exploration of StarCraft map space 

» Bin Tan/Theo/Anthony 2010: evolution of neural Go players 

» Preuss/Quadflieg/Rudolph 2011: multi-objective track selection 


Q We may have missed some, but still... 


Multi-objective representative selection 


Q Small number of driver configurations that add up well for 
different tracks (Preuss/Quadflieg/Rudolph, 2011) 


» Full MO run with 6 objectives very expensive (runtime, 
instability) 


» Single-objective on 2 tracks: very different solutions 
» We evaluate the best solutions of both on all tracks 
» Correlation analysis (rank based): similarity of tracks 
» Simple and working, but much to do here... 


Wheelz 4 19 
os 

Strest-1 4 
os 
CG-Speedway-1 ni 
C-SpeedWay 4 02 
Brondehach -| 00 
-02 
-04 
rsum -] eo 

as» 2 


upper: p values corr test 


é 


lower. tau correlations 


POLIMI 


GECCO, July 6-10, 2013, Amsterdam, NL 


968 


Many objectives: camera positioning 


Q At first: weighted single-objective (Preuss/Burelli/Yannakakis, 
2012) 


» Vision: realtime multi-objective (often 3-9), diverse solutions 
» Currently not possible (time), but learned a lot about problem 
» Very interesting benchmark for optimization methods 


Evolving FPS Maps: overview 


Q Cardamone et al. evolved 
maps for CUBE, an open 
source First Person Shooter 

Q Four different 
representations 
were compared 

Q Fitness based on game 
statistics computed using 
NPCs 
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Evolving maps for FPS 


Evolving FPS Maps: examples 


Evolving Racing Tracks 


Evolving Tracks: examples 
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Evolving Racing Tracks: the approach 


Q Apply a genetic algorithm to evolve the shape of the tracks 
for a realistic racing game ` 


QO Representation based on a list of YV 
way points in polar coordinates N 


Q Fitness based on diversity: 
» Entropy of speeds 
» Entropy of curvatures 
» Both 


VS 


How to put users in the loop? 


A couple of examples... 
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Galactic Arms Race (GAR) 


http://gar.eecs.ucf.edu/ 


http://eplex.cs.ucf.edu/movies/gar_promo2.wmv 
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TORCS/Speed Dreams Tracks Generator 


http://trackgen.pierlucalanzi.net 


http://youtu.be/YFOa7L30BwM 
http://youtu.be/O W4jHN2h2Q 
http://youtu.be/Si u 43HJnM 


http://youtu.be/3AzjMtRDnBo 
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Racing Games 


Point-to-point 
RARS 
TORCS / Speed Dreams 
VDrift 
Simulated Car Racing 
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Point-to-point car racing and RARS 


Point-to-point RARS 


EX EITTIT 


Q Open source 3D racing 
simulator 

Q Designed to enabled pre- 
programmed AI drivers to 
race against 

O http://rars.sourceforge.net/ 


us 


Q Simple Java racing 
environment (Togelius et 
al., 2005) 

C] Used for competitions at CIG 
2007 and at CEC 2007 

QO http://julian.togelius.com/ 
cec2007competition/ 


Simulated Car Racing 


Q Simulated Car Racing (SCR) requires the development of a 
driver for TORCS (hand-coded, learned, evolved, ...) 


Q SCR typically involves 9 races organized in three different 
legs during three major conferences 


Q Teams are awarded based on their score in each conference 
competition 


U At the end, the team with highest overall score wins the 
championship 


Q SCR has been organized since 2009 


2 http://games.ws.dei.polimi.it/competitions/scr/ 
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TORCS / Speed Dreams and VDrift 


TORCS / Speed Dreams VDrift 


Q Use Bullet, an open source 
physics engine featuring 3D 
collision detection, soft and 
rigid body dynamics 

Q Accurate simulation of loss 
of traction (drift) 


Q Accurate physics engine 
specifically developed for 
racing (traction, damage, 
aerodynamics,...) 

O Wide community of users 

providing tracks and other 

game content U http: 


vdrift.net 


QO) http://torcs.sourceforge.net 


Simulated Car Racing: architecture 


ee, 


TORCS 


ee 


PATCH 


SBOT§SBOT §SBOT 
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Simulated Car Racing: sensors and 
actuators 


Q Rangefinders for edges on the track and opponents (with noise) 
Q Speed, RPM, fuel, damage, angle with track, distance race, position 
on track, etc. 


u—MÁ—À 


First Person Shooters 


ioquake3 
Cube 
Unreal Tournament 


i [^ FA x 2K Bot Prize 


Q Six effectors: steering wheel [-1,+1], gas pedal [0, +1], brake 
pedal [0,+1], gearbox {-1,0,1,2,3,4,5,6}, clutch [0,+1], focus 
direction 
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ioquake3 and Cube Unreal Tournament 


ioquake3 
Q Very popular series of multiplayer FPS by Epic Games 


Q Does not require expensive hardware to run 
C] Can be easily customized with scripting 

QO) http://www.unrealtournament.com/ 

C) Unreal Wiki: http://wiki.beyondunreal.com/ 


Q Bug-free and enhanced ū Cube 2: Sauerbraten is a 


implementation of the id free multiplayer/singleplayer 
Software's Quake 3 engine first person shooter 
a Used in several game Q Allow map/geometry editing 
projects as well as in several to be done dynamically in- 
academic projects game 
Q http://ioquake3.org Q http://sauerbraten.org/ 
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2K BotPrize 


QO The BotPrize competition 
challenges programmers/ 
researchers/hobbyists to 
create a bot for UT2004 (a 
first-person shooter) that 
can fool opponents into 
thinking it is another human 
player. 

Q The competition organized 
by P. Hingston has been 
sponsored by 2K games 
since 2008, and the $5000 
major prize is yet to be 
claimed. 

Q http://www.botprize.org/ 


Starcraft Competition 


Q Held at AIIDE and CIG 
conferences since 2010, setup 
differs slightly: AIIDE maps are 
known beforehand, CIG maps 

Q Bots attached to Starcraft via 3'd 
person hack BWAPI: 
http://code.google.com/p/bwapi/ 

Q) Active scene of around 20 bot 
developers/teams 

Q Both competitions won by Skynet 
bot in 2011 

Q Current limitations: most bots are 
not very adaptive to opponent 
strategy 


http://youtu.be/xXsx1ma3_ko 


http://webdocs.cs.ualberta.ca/~cdavid/starcraftaicomp 
http://Is11-www.cs.uni-dortmund.de/rts-competition/starcraft-cig2012 
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Strategy Games 


Starcraft and Starcraft Competition 
Stargus and Wargus 


Stratagus, Stargus and Wargus 


Q Stratagus is a free cross- 
platform real-time strategy 
gaming engine. 

O It includes support for playing 
over the internet/LAN, or 
playing a computer opponent. 

O It is easily configurable and can 
be used to create games with a 
wide-range of features specific 
to your needs. 

Q Stargus and Wargus are mods 

that allow to play the popular 

Starcraft and Warcraft games 

with Stratugus engine 

http://stratagus.com/ 
http://wargus.sourceforge.net/ 
http://stargus.sourceforge.net/ 


ooo 


Robocode 


QO Robocode is a programming game, where the goal is to 
develop a robot battle tank to battle against other tanks 
Q The robot tanks can be developed either in Java or .NET. 
QO Battles can be either run in real-time and displayed on the 


Action Games screen or run in a batch mode without visualization. 
O It has a large community and features an on-line tournament 
Robocode system to rank developed tanks 


O Official page: http://robocode.sourceforge.net/ 
O Robo wiki: http://robowiki.net, 


Infinite Mario 
Ms. Pac-Man 
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Infinite Mario Mario AI competition 
Q Infinite Mario Bros is a Java-based browser game developed Q Mario AI competition has 

by Markus Persson for Super Mario themed contest been organized by Togelius 
Q It provides unending 2D platforming action: all areas and et al since 2009 as part of 2i dpgrisuurt rias 

level selection maps are generated from a random seed. the major conferences of the ‘ 
Q The game and the entire source code is available for field 

download at http://www.mojang.com/notch/mario/ Q It consists of four tracks 

with different goals: 
» Gameplay 


» Learning 
» Level Generation 
» Turing Test 


Q Competitors are provided 
with very effective Java APIs 
based on Mario Infinite 
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Ms. Pac-Man 


Q A sequel of Pac-Man with a 
very similar gameplay 

Q It features four different mazes 

Q Involves non-deterministic 
opponents 

Q http://www.webpacman.com/ 
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Develop your own game framework 


XNA 
Unity 
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Ms. Pac-Man competitions 


Q Ms. Pac-man vs Ghosts Competition (Rohlfshagen, Robles 
and Lucas) 


» allows you to develop AI controllers for either Ms Pac-Man 
or for the ghosts 


» based on a powerful Java framework developed by the 
organizers 


» http://www.pacman-vs-ghosts.net/ 
Q Ms Pac-Man Screen Capture Competition (Lucas) 

» The aim of this competition is to provide the best software 
controller for the game of Ms Pac-Man. 

» It is based on the original version of the game 

» About 15 times per second the controller receives a pixel 
map of the Ms. Pac-Man window and it has to respond 
with the direction of the joystick. 

» http://cswww.essex.ac.uk/staff/sml/pacman/ 
PacManContest.html 
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XNA 


Q Framework developed by 
Microsoft to simplify the 
development of games 

OQ The development with XNA 
is performed in C£ 
under .NET framework 

a Target platforms: 

» Windows 
» Windows Phone 
» Xbox 360 
Q Free 
QO http://create.msdn.com 
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Unity 3D 


Q A complete game engine for 
3d games that features: 
» Physics engine 
» Camera control 
» Animation system 
b. 

Q Allows development on 
Windows and Mac both in 
C# and in Javascript 

Q Target platforms: 

» Windows and Mac 

» Mobile (Android, iOS) 
» Web 

» Consoles 

Q Offers free licenses with 
limitations and paid licenses 

Q http://www.unity3d.com 


HOT topics / Current Trends 


Q Believability of NSC and their environment 
» More humanlike behavior 
» Better cooperation of units (team AI) 
» Reactivity to unforeseen events 
QO Personalization of games 
» Preference modeling (what do they like?) 
» Player type analysis, classification 
» Dynamic adaptation of game mechanisms (e.g. difficulty) 
Q Procedural Content Generation 
» Offline to support game creation 
» Online to enlarge worlds 
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Conclusions 


Where are we going? 


Q The testbed argument seems to loose importance: 


» test problem collections (benchmarks) and competitions are 
getting popular in many fields (e.g. BBOB) 


» not really simple to transfer back obtained knowledge (games 
research partly engineering) 


» the need to defend games research is shrinking 


O The doing things better argument is (still) important: 
» Can involve theory, but usually based on experimentation 
» Question: what does better mean? 


» Measurement sometimes fully automated, sometimes requires 
user interaction (no fun formula) 


Required: being open to other methods (to achieve meaningful 
comparisons) 


Ideal situation: competition as joint effort experiment (fair) 


Y 


há 
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To boldly go... 


a We may encounter unsolved or even unrealized problems: 
» Interesting features of CI techniques: surprising solutions, 
highly adaptable to problem, multiple objectives 
» Show that our approach indeed does fulfill some minimal 
requirements by experiment 


p" 


Takehome message 


Q CI tools, especially Evolutionary Algorithms are well suited 
for many applications in Games 

Q Multi-objective EA very useful, but rarely used 

Q Some very dynamic areas identified: PCG, Personalization 

Q Lots of possibilities to enter the arena: competitions, free 
engines, etc. 

Q Game based benchmarks good to motivate people (students) 
and to showcase your research 


Q Newly established web base for CI/AI game research 
projects and demos: http://www.aigameresearch.org/ 
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Algorithm development and theory 


Q Of course we can improve our methods while applying them 

Q But this is usually not unique for games problems 

Q Improvement/improved understanding may result in better 
theory 

Q Discrete state games: algorithm engineering cycle 
applicable 

Q More complex games (e.g. RTS): theory connection very 
difficult 

Q Solving games problems is to a large extent engineering 

Q We have to rely on good experimentation in most cases 


Thank you! 


